Best Practices for Service Segmentation Based On Environment

#1

Hey there, I’ve been searching the web and forums for any advice on best practices for service setup with respect to segmenting across different environments.

For example, we have a beta, gamma, and production environment which basically mirror each other but the production environment is serving real traffic (your typical setup).

How do folks typically setup pagerduty in this situation? Seems like teams would want to be able to manage incidents with the knowledge of what environment they are in and branch different workflows based on the environment.

For example

  • Do people create one service for each environment? i.e. Beta is a service, Gamma is a service, and Production is a service. Then create one service for each actual piece of infrastructure? So if our project has say an API service, we would create a service in PagerDuty called API?
  • Or maybe people typically create 3 services (continuing from the example of above): API - Beta, API - Gamma, API - Prod?
  • Or perhaps PagerDuty best practice is to only instrument the production environment so there is no use in segmenting by environment at all?

Please help! Which approach is best?

#2

I’m always curious how other PD users handle this :thinking:

Where I work, we create two PagerDuty services per entry needed from our internal component registry. If we spun up a front-end API we’d have, in PagerDuty, a “Front-End API Prod” and “Front-End API NonProd” service. All production alerts route to the prod version and all nonproduction alerts route to the nonproduction. (uat, dev, test, beta, etc)

This pattern holds up well with our workflow as nonproduction environments incidents default to low urgency and alerts are often suppressed outside of business hours. We use them to spot issues but we aren’t waking up to fix things at 02:00 in the morning. It gives just enough targets to split alert handling without drowning the dev teams in choices.

Prod services, of course, default to lighting the beacons of Gondor and waking you from a dead sleep in the cold of the night without remorse.

Personally, I rather like a single service target and using event orchestration with alert transformations. Each alert comes in, hits an orchestration path based on source (environment), titles are updated to reflect the environment, and any additional transformations are handled. It’s all in one place which I find easier to manage. This last bit is just a personal preference, though, and based on a small scale (15-30 services).